{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rvpBDcdS5TSR"
   },
   "source": [
    "# *fairlib*: A Unified Framework for Assessing and Improving Fairness\n",
    "\n",
    "Xudong Han$^{1}$, &nbsp; Aili Shen$^{1,2, a}$, &nbsp; Yitong Li$^{3}$, &nbsp; Lea Frermann$^{1}$, &nbsp; Timothy Baldwin$^{1,4}$, &nbsp; and &nbsp; Trevor Cohn$^{1}$  \n",
    "\n",
    "$^{1}$ The University of Melbourne\n",
    "\n",
    "$^{2}$ Alexa AI, Amazon\n",
    "\n",
    "$^{3}$ Huawei Technologies Co., Ltd.\n",
    "\n",
    "$^{4}$ MBZUAI\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/en/thumb/e/ed/Logo_of_the_University_of_Melbourne.svg/330px-Logo_of_the_University_of_Melbourne.svg.png\" height=\"100\"/> &nbsp; &nbsp; &nbsp; &nbsp;\n",
    "<img src=\"https://2019.emnlp.org/assets/images/logos/huawei-logo.png\" height=\"100\"/> &nbsp; &nbsp; &nbsp; &nbsp;\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/en/5/55/Mohamed_bin_Zayed_University_of_Artificial_Intelligence_logo.png\" height=\"100\"/>\n",
    "\n",
    "---\n",
    "- $^a$ Work carried out at The University of Melbourne\n",
    "- *fairlib* is licensed under the **Apache License 2.0**\n",
    "\n",
    "[GitHub](https://github.com/HanXudong/fairlib), [Docs](https://hanxudong.github.io/fairlib/), [PyPI](https://pypi.org/project/fairlib/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uwB4au4ievWx"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HanXudong/fairlib/blob/main/tutorial/fairlib_demo.ipynb)\n",
    "\n",
    "In this video, we will demostrate how to:\n",
    "1.   Install *fairlib*\n",
    "2.   Access fairness benchmark datasets\n",
    "3.   Train a vanilla model without debiasing, and measure fairness\n",
    "4.   Improve fairness with most recent debiasing methods\n",
    "5.   Analyze the results, such as creating tables and figures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "xIzDL6q3hm2b"
   },
   "outputs": [],
   "source": [
    "import fairlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Unexpected args: ['-f', '/beegfs/home/artem.vazhentsev/.local/share/jupyter/runtime/kernel-fb2be37b-b00e-4e3d-8924-24703227c52c.json']\n",
      "INFO:root:Logging to ./fairlib/results/dev/Rob_gender/test/output.log\n",
      "WARNING:root:Log file already exists, will append\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:10:58 [INFO ]  ======================================== 2023-03-16 18:10:58 ========================================\n",
      "2023-03-16 18:10:58 [INFO ]  Base directory is ./fairlib/results/dev/Rob_gender/test\n",
      "2023-03-16 18:10:58 [INFO ]  Options: \n",
      "2023-03-16 18:10:58 [INFO ]  \tBT: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tBTObj: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tDyBT: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tDyBTObj: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tDyBTalpha: 0.1\n",
      "2023-03-16 18:10:58 [INFO ]  \tDyBTinit: original\n",
      "2023-03-16 18:10:58 [INFO ]  \tFCL: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tFCLObj: g\n",
      "2023-03-16 18:10:58 [INFO ]  \tGAP_metric_name: TPR_GAP\n",
      "2023-03-16 18:10:58 [INFO ]  \tGBT: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tGBTObj: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tGBT_N: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tGBT_alpha: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tINLP: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tINLP_by_class: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tINLP_discriminator_reweighting: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tINLP_min_acc: 0.0\n",
      "2023-03-16 18:10:58 [INFO ]  \tINLP_n: 190\n",
      "2023-03-16 18:10:58 [INFO ]  \tPerformance_metric_name: accuracy\n",
      "2023-03-16 18:10:58 [INFO ]  \tactivation_function: Tanh\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_BT: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_BTObj: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_activation_function: ReLu\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_batch_norm: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_batch_size: 1024\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_checkpoint_interval: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_debiasing: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_decoupling: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_decoupling_labelled_proportion: 1.0\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_diverse_lambda: 0.0\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_dropout: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_epochs: 100\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_epochs_since_improvement: 5\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_gated: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_gated_mapping: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_hidden_size: 300\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_lambda: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_level: last_hidden\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_lr: 0.001\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_n_hidden: 2\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_num_subDiscriminator: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_test_batch_size: 1024\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_uniform_label: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tadv_update_frequency: Batch\n",
      "2023-03-16 18:10:58 [INFO ]  \tbase_seed: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tbatch_norm: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tbatch_size: 32\n",
      "2023-03-16 18:10:58 [INFO ]  \tcheckpoint_dir: models\n",
      "2023-03-16 18:10:58 [INFO ]  \tcheckpoint_interval: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tcheckpoint_name: BEST_checkpoint\n",
      "2023-03-16 18:10:58 [INFO ]  \tclassification_head_update_frequency: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tconf_file: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tdata_dir: ../../../../datasets/RoB/sex\n",
      "2023-03-16 18:10:58 [INFO ]  \tdataset: Rob_gender\n",
      "2023-03-16 18:10:58 [INFO ]  \tdevice_id: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tdropout: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tearly_stopping_criterion: loss\n",
      "2023-03-16 18:10:58 [INFO ]  \temb_size: 768\n",
      "2023-03-16 18:10:58 [INFO ]  \tencoder_architecture: BERT\n",
      "2023-03-16 18:10:58 [INFO ]  \tepochs: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tepochs_since_improvement: 5\n",
      "2023-03-16 18:10:58 [INFO ]  \texp_id: test\n",
      "2023-03-16 18:10:58 [INFO ]  \tf: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_base_temperature_g: 0.01\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_base_temperature_y: 0.01\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_lambda_g: 0.1\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_lambda_y: 0.1\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_temperature_g: 0.01\n",
      "2023-03-16 18:10:58 [INFO ]  \tfcl_temperature_y: 0.01\n",
      "2023-03-16 18:10:58 [INFO ]  \tfull_label: true\n",
      "2023-03-16 18:10:58 [INFO ]  \tgated: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tgated_mapping: null\n",
      "2023-03-16 18:10:58 [INFO ]  \tgroup_agg_power: null\n",
      "2023-03-16 18:10:58 [INFO ]  \thidden_size: 300\n",
      "2023-03-16 18:10:58 [INFO ]  \tlog_file: ./fairlib/results/dev/Rob_gender/test/output.log\n",
      "2023-03-16 18:10:58 [INFO ]  \tlog_interval: 50\n",
      "2023-03-16 18:10:58 [INFO ]  \tlog_level: INFO\n",
      "2023-03-16 18:10:58 [INFO ]  \tlr: 0.003\n",
      "2023-03-16 18:10:58 [INFO ]  \tmodel_dir: ./fairlib/results/dev/Rob_gender/test/models\n",
      "2023-03-16 18:10:58 [INFO ]  \tn_bins: 4\n",
      "2023-03-16 18:10:58 [INFO ]  \tn_freezed_layers: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tn_hidden: 2\n",
      "2023-03-16 18:10:58 [INFO ]  \tn_jobs: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tn_power_iterations: 1\n",
      "2023-03-16 18:10:58 [INFO ]  \tno_log: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tnum_classes: 2\n",
      "2023-03-16 18:10:58 [INFO ]  \tnum_clusters: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tnum_groups: 2\n",
      "2023-03-16 18:10:58 [INFO ]  \tnum_remove_clusters: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tnum_workers: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tproject_dir: dev\n",
      "2023-03-16 18:10:58 [INFO ]  \tregression: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tremove_percent: 0\n",
      "2023-03-16 18:10:58 [INFO ]  \tresults_dir: ./fairlib/results\n",
      "2023-03-16 18:10:58 [INFO ]  \tsave_batch_results: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tselection_criterion: accuracy\n",
      "2023-03-16 18:10:58 [INFO ]  \tstart_time: '2023-03-16 18:10:58'\n",
      "2023-03-16 18:10:58 [INFO ]  \tsubsample_perc: 1.0\n",
      "2023-03-16 18:10:58 [INFO ]  \ttest_batch_size: 32\n",
      "2023-03-16 18:10:58 [INFO ]  \tuse_skipconnection: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tuse_spectralnorm: false\n",
      "2023-03-16 18:10:58 [INFO ]  \tweight_decay: 0.0\n",
      "2023-03-16 18:10:58 [INFO ]  \t\n",
      "2023-03-16 18:10:58 [WARNING]  ./fairlib/results/dev/Rob_gender/test/opt.yaml already exists, moved to ./fairlib/results/dev/Rob_gender/test/old_opts/opt_2023_03_16__18_09_03.yaml\n",
      "Loaded data shapes: (13825, 128), (13825,), (13825,)\n",
      "Loaded data shapes: (1536, 128), (1536,), (1536,)\n",
      "Loaded data shapes: (1707, 128), (1707,), (1707,)\n",
      "2023-03-16 18:11:29 [INFO ]  train dataset size:\t13825\n",
      "2023-03-16 18:11:29 [INFO ]  validation dataset size: \t1536\n",
      "2023-03-16 18:11:29 [INFO ]  test dataset size: \t1707\n",
      "2023-03-16 18:11:29 [INFO ]  datasets built!\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "import logging\n",
    "\n",
    "from fairlib.src.base_options import BaseOptions\n",
    "from fairlib.src import networks\n",
    "\n",
    "args = {\n",
    "    # The name of the dataset, corresponding dataloader will be used,\n",
    "    \"dataset\":  \"Rob_gender\",\n",
    "\n",
    "    # Specifiy the path to the input data\n",
    "    \"data_dir\": \"../../../../datasets/RoB/sex\",\n",
    "\n",
    "    # Device for computing, -1 is the cpu; non-negative numbers indicate GPU id.\n",
    "    \"device_id\":    0,\n",
    "    \n",
    "    \"emb_size\": 768,\n",
    "    \n",
    "    \"encoder_architecture\": \"BERT\",\n",
    "    \n",
    "    \"epochs\": 1,\n",
    "\n",
    "    # The default path for saving experimental results\n",
    "    \"results_dir\":  r\"./fairlib/results\",\n",
    "\n",
    "    # Will be used for saving experimental results\n",
    "    \"project_dir\":  r\"dev\",\n",
    "\n",
    "    # We will focusing on TPR GAP, implying the Equalized Odds for binary classification.\n",
    "    \"GAP_metric_name\":  \"TPR_GAP\",\n",
    "\n",
    "    # The overall performance will be measured as accuracy\n",
    "    \"Performance_metric_name\":  \"accuracy\",\n",
    "    # Model selections are based on distance to optimum, see section 4 in our paper for more details\n",
    "    \"selection_criterion\":  \"accuracy\",\n",
    "\n",
    "    # Default dirs for saving checkpoints\n",
    "    \"checkpoint_dir\":   \"models\",\n",
    "    \"checkpoint_name\":  \"BEST_checkpoint\",\n",
    "    \"test_batch_size\":32,\n",
    "    \"batch_size\":32,\n",
    "\n",
    "    # Loading experimental results\n",
    "    \"n_jobs\":   1,\n",
    "}\n",
    "\n",
    "options = BaseOptions()\n",
    "state = options.get_state(args=args)\n",
    "\n",
    "# Init the model\n",
    "#model = networks.get_main_model(state)\n",
    "\n",
    "#model.train_self()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(13825, 128)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state.opt.train_generator.dataset.X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1536, 128)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state.opt.dev_generator.dataset.X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import KFold\n",
    "X = np.concatenate([state.opt.train_generator.dataset.X, state.opt.dev_generator.dataset.X])[:100]\n",
    "y = np.concatenate([state.opt.train_generator.dataset.y, state.opt.dev_generator.dataset.y])[:100]\n",
    "protected_label = np.concatenate([state.opt.train_generator.dataset.protected_label, state.opt.dev_generator.dataset.protected_label])[:100]\n",
    "\n",
    "if state.encoder_architecture==\"BERT\":\n",
    "    token_type_ids = np.concatenate([state.opt.train_generator.dataset.token_type_ids, state.opt.dev_generator.dataset.token_type_ids])[:100]\n",
    "    mask = np.concatenate([state.opt.train_generator.dataset.mask, state.opt.dev_generator.dataset.mask])[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((100, 128), (100, 128))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, mask.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "def gen_data(a=[1,2,3,4,5]):\n",
    "    for x in a:\n",
    "        yield x\n",
    "        \n",
    "        \n",
    "for x in gen_data():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:11:44 [INFO ]  MLP( \n",
      "2023-03-16 18:11:44 [INFO ]    (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]    (AF): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]    (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]    (hidden_layers): ModuleList(\n",
      "2023-03-16 18:11:44 [INFO ]      (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]      (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]      (2): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]      (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]      (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]      (5): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]    )\n",
      "2023-03-16 18:11:44 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:11:44 [INFO ]  )\n",
      "2023-03-16 18:11:44 [INFO ]  Total number of parameters: 321602 \n",
      "\n",
      "2023-03-16 18:11:44 [INFO ]  BERTClassifier( \n",
      "2023-03-16 18:11:44 [INFO ]    (bert): BertModel(\n",
      "2023-03-16 18:11:44 [INFO ]      (embeddings): BertEmbeddings(\n",
      "2023-03-16 18:11:44 [INFO ]        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "2023-03-16 18:11:44 [INFO ]        (position_embeddings): Embedding(512, 768)\n",
      "2023-03-16 18:11:44 [INFO ]        (token_type_embeddings): Embedding(2, 768)\n",
      "2023-03-16 18:11:44 [INFO ]        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]        (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]      )\n",
      "2023-03-16 18:11:44 [INFO ]      (encoder): BertEncoder(\n",
      "2023-03-16 18:11:44 [INFO ]        (layer): ModuleList(\n",
      "2023-03-16 18:11:44 [INFO ]          (0): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (1): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (2): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (3): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (4): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (5): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (6): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (7): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (8): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (9): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (10): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]          (11): BertLayer(\n",
      "2023-03-16 18:11:44 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:11:44 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:11:44 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:11:44 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]              )\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:11:44 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:11:44 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]            )\n",
      "2023-03-16 18:11:44 [INFO ]          )\n",
      "2023-03-16 18:11:44 [INFO ]        )\n",
      "2023-03-16 18:11:44 [INFO ]      )\n",
      "2023-03-16 18:11:44 [INFO ]      (pooler): BertPooler(\n",
      "2023-03-16 18:11:44 [INFO ]        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]        (activation): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]      )\n",
      "2023-03-16 18:11:44 [INFO ]    )\n",
      "2023-03-16 18:11:44 [INFO ]    (classifier): MLP(\n",
      "2023-03-16 18:11:44 [INFO ]      (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]      (AF): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]      (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]      (hidden_layers): ModuleList(\n",
      "2023-03-16 18:11:44 [INFO ]        (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]        (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]        (2): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]        (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:11:44 [INFO ]        (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:11:44 [INFO ]        (5): Tanh()\n",
      "2023-03-16 18:11:44 [INFO ]      )\n",
      "2023-03-16 18:11:44 [INFO ]      (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:11:44 [INFO ]    )\n",
      "2023-03-16 18:11:44 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:11:44 [INFO ]  )\n",
      "2023-03-16 18:11:44 [INFO ]  Total number of parameters: 109803842 \n",
      "\n",
      "Number of trainable parameters: 109482240\n",
      "2023-03-16 18:11:47 [INFO ]  Epoch:    0 [      0/     80 ( 0%)]\tLoss: 0.6927\t Data Time: 0.03s\tTrain Time: 3.08s\n",
      "2023-03-16 18:11:47 [INFO ]  Loss, accuracy and DTO: 4.934114 0.200000 0.874644\n",
      "2023-03-16 18:11:47 [INFO ]  Evaluation at Epoch 0\n",
      "2023-03-16 18:11:58 [INFO ]  Validation accuracy: 20.00\tmacro_fscore: 16.67\tmicro_fscore: 20.00\tTPR_GAP: 35.36\tFPR_GAP: 70.71\tPPR_GAP: 0.00\t\n",
      "2023-03-16 18:11:58 [INFO ]  Test accuracy: 27.83\tmacro_fscore: 21.77\tmicro_fscore: 27.83\tTPR_GAP: 0.00\tFPR_GAP: 0.00\tPPR_GAP: 0.00\t\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:12:03 [INFO ]  MLP( \n",
      "2023-03-16 18:12:03 [INFO ]    (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]    (AF): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]    (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]    (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:03 [INFO ]      (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]      (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]      (2): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]      (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]      (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]      (5): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]    )\n",
      "2023-03-16 18:12:03 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:03 [INFO ]  )\n",
      "2023-03-16 18:12:03 [INFO ]  Total number of parameters: 321602 \n",
      "\n",
      "2023-03-16 18:12:03 [INFO ]  BERTClassifier( \n",
      "2023-03-16 18:12:03 [INFO ]    (bert): BertModel(\n",
      "2023-03-16 18:12:03 [INFO ]      (embeddings): BertEmbeddings(\n",
      "2023-03-16 18:12:03 [INFO ]        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "2023-03-16 18:12:03 [INFO ]        (position_embeddings): Embedding(512, 768)\n",
      "2023-03-16 18:12:03 [INFO ]        (token_type_embeddings): Embedding(2, 768)\n",
      "2023-03-16 18:12:03 [INFO ]        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]        (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]      )\n",
      "2023-03-16 18:12:03 [INFO ]      (encoder): BertEncoder(\n",
      "2023-03-16 18:12:03 [INFO ]        (layer): ModuleList(\n",
      "2023-03-16 18:12:03 [INFO ]          (0): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (1): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (2): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (3): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (4): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (5): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (6): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (7): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (8): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (9): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (10): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]          (11): BertLayer(\n",
      "2023-03-16 18:12:03 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:03 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:03 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:03 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]              )\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:03 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:03 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]            )\n",
      "2023-03-16 18:12:03 [INFO ]          )\n",
      "2023-03-16 18:12:03 [INFO ]        )\n",
      "2023-03-16 18:12:03 [INFO ]      )\n",
      "2023-03-16 18:12:03 [INFO ]      (pooler): BertPooler(\n",
      "2023-03-16 18:12:03 [INFO ]        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]        (activation): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]      )\n",
      "2023-03-16 18:12:03 [INFO ]    )\n",
      "2023-03-16 18:12:03 [INFO ]    (classifier): MLP(\n",
      "2023-03-16 18:12:03 [INFO ]      (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]      (AF): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]      (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]      (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:03 [INFO ]        (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]        (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]        (2): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]        (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:03 [INFO ]        (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:03 [INFO ]        (5): Tanh()\n",
      "2023-03-16 18:12:03 [INFO ]      )\n",
      "2023-03-16 18:12:03 [INFO ]      (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:03 [INFO ]    )\n",
      "2023-03-16 18:12:03 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:03 [INFO ]  )\n",
      "2023-03-16 18:12:03 [INFO ]  Total number of parameters: 109803842 \n",
      "\n",
      "Number of trainable parameters: 109482240\n",
      "2023-03-16 18:12:03 [INFO ]  Epoch:    0 [      0/     80 ( 0%)]\tLoss: 0.6602\t Data Time: 0.00s\tTrain Time: 0.40s\n",
      "2023-03-16 18:12:04 [INFO ]  Loss, accuracy and DTO: 0.512550 0.800000 0.538519\n",
      "2023-03-16 18:12:04 [INFO ]  Evaluation at Epoch 0\n",
      "2023-03-16 18:12:15 [INFO ]  Validation accuracy: 80.00\tmacro_fscore: 44.44\tmicro_fscore: 80.00\tTPR_GAP: 50.00\tFPR_GAP: 158.12\tPPR_GAP: 158.11\t\n",
      "2023-03-16 18:12:15 [INFO ]  Test accuracy: 72.17\tmacro_fscore: 41.92\tmicro_fscore: 72.17\tTPR_GAP: 0.00\tFPR_GAP: 0.00\tPPR_GAP: 0.00\t\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:12:19 [INFO ]  MLP( \n",
      "2023-03-16 18:12:19 [INFO ]    (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]    (AF): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]    (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]    (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:19 [INFO ]      (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]      (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]      (2): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]      (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]      (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]      (5): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]    )\n",
      "2023-03-16 18:12:19 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:19 [INFO ]  )\n",
      "2023-03-16 18:12:19 [INFO ]  Total number of parameters: 321602 \n",
      "\n",
      "2023-03-16 18:12:19 [INFO ]  BERTClassifier( \n",
      "2023-03-16 18:12:19 [INFO ]    (bert): BertModel(\n",
      "2023-03-16 18:12:19 [INFO ]      (embeddings): BertEmbeddings(\n",
      "2023-03-16 18:12:19 [INFO ]        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "2023-03-16 18:12:19 [INFO ]        (position_embeddings): Embedding(512, 768)\n",
      "2023-03-16 18:12:19 [INFO ]        (token_type_embeddings): Embedding(2, 768)\n",
      "2023-03-16 18:12:19 [INFO ]        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]        (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]      )\n",
      "2023-03-16 18:12:19 [INFO ]      (encoder): BertEncoder(\n",
      "2023-03-16 18:12:19 [INFO ]        (layer): ModuleList(\n",
      "2023-03-16 18:12:19 [INFO ]          (0): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (1): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (2): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (3): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (4): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (5): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (6): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (7): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (8): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (9): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (10): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]          (11): BertLayer(\n",
      "2023-03-16 18:12:19 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:19 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:19 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:19 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]              )\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:19 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:19 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]            )\n",
      "2023-03-16 18:12:19 [INFO ]          )\n",
      "2023-03-16 18:12:19 [INFO ]        )\n",
      "2023-03-16 18:12:19 [INFO ]      )\n",
      "2023-03-16 18:12:19 [INFO ]      (pooler): BertPooler(\n",
      "2023-03-16 18:12:19 [INFO ]        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]        (activation): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]      )\n",
      "2023-03-16 18:12:19 [INFO ]    )\n",
      "2023-03-16 18:12:19 [INFO ]    (classifier): MLP(\n",
      "2023-03-16 18:12:19 [INFO ]      (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]      (AF): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]      (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]      (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:19 [INFO ]        (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]        (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]        (2): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]        (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:19 [INFO ]        (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:19 [INFO ]        (5): Tanh()\n",
      "2023-03-16 18:12:19 [INFO ]      )\n",
      "2023-03-16 18:12:19 [INFO ]      (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:19 [INFO ]    )\n",
      "2023-03-16 18:12:19 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:19 [INFO ]  )\n",
      "2023-03-16 18:12:19 [INFO ]  Total number of parameters: 109803842 \n",
      "\n",
      "Number of trainable parameters: 109482240\n",
      "2023-03-16 18:12:20 [INFO ]  Epoch:    0 [      0/     80 ( 0%)]\tLoss: 0.6764\t Data Time: 0.00s\tTrain Time: 0.39s\n",
      "2023-03-16 18:12:20 [INFO ]  Loss, accuracy and DTO: 0.536689 0.800000 0.538519\n",
      "2023-03-16 18:12:20 [INFO ]  Evaluation at Epoch 0\n",
      "2023-03-16 18:12:31 [INFO ]  Validation accuracy: 80.00\tmacro_fscore: 44.44\tmicro_fscore: 80.00\tTPR_GAP: 50.00\tFPR_GAP: 158.12\tPPR_GAP: 158.11\t\n",
      "2023-03-16 18:12:31 [INFO ]  Test accuracy: 72.17\tmacro_fscore: 41.92\tmicro_fscore: 72.17\tTPR_GAP: 0.00\tFPR_GAP: 0.00\tPPR_GAP: 0.00\t\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:12:36 [INFO ]  MLP( \n",
      "2023-03-16 18:12:36 [INFO ]    (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]    (AF): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]    (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]    (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:36 [INFO ]      (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]      (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]      (2): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]      (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]      (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]      (5): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]    )\n",
      "2023-03-16 18:12:36 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:36 [INFO ]  )\n",
      "2023-03-16 18:12:36 [INFO ]  Total number of parameters: 321602 \n",
      "\n",
      "2023-03-16 18:12:36 [INFO ]  BERTClassifier( \n",
      "2023-03-16 18:12:36 [INFO ]    (bert): BertModel(\n",
      "2023-03-16 18:12:36 [INFO ]      (embeddings): BertEmbeddings(\n",
      "2023-03-16 18:12:36 [INFO ]        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "2023-03-16 18:12:36 [INFO ]        (position_embeddings): Embedding(512, 768)\n",
      "2023-03-16 18:12:36 [INFO ]        (token_type_embeddings): Embedding(2, 768)\n",
      "2023-03-16 18:12:36 [INFO ]        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]        (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]      )\n",
      "2023-03-16 18:12:36 [INFO ]      (encoder): BertEncoder(\n",
      "2023-03-16 18:12:36 [INFO ]        (layer): ModuleList(\n",
      "2023-03-16 18:12:36 [INFO ]          (0): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (1): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (2): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (3): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (4): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (5): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (6): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (7): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (8): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (9): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (10): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]          (11): BertLayer(\n",
      "2023-03-16 18:12:36 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:36 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:36 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:36 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]              )\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:36 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:36 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]            )\n",
      "2023-03-16 18:12:36 [INFO ]          )\n",
      "2023-03-16 18:12:36 [INFO ]        )\n",
      "2023-03-16 18:12:36 [INFO ]      )\n",
      "2023-03-16 18:12:36 [INFO ]      (pooler): BertPooler(\n",
      "2023-03-16 18:12:36 [INFO ]        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]        (activation): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]      )\n",
      "2023-03-16 18:12:36 [INFO ]    )\n",
      "2023-03-16 18:12:36 [INFO ]    (classifier): MLP(\n",
      "2023-03-16 18:12:36 [INFO ]      (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]      (AF): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]      (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]      (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:36 [INFO ]        (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]        (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]        (2): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]        (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:36 [INFO ]        (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:36 [INFO ]        (5): Tanh()\n",
      "2023-03-16 18:12:36 [INFO ]      )\n",
      "2023-03-16 18:12:36 [INFO ]      (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:36 [INFO ]    )\n",
      "2023-03-16 18:12:36 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:36 [INFO ]  )\n",
      "2023-03-16 18:12:36 [INFO ]  Total number of parameters: 109803842 \n",
      "\n",
      "Number of trainable parameters: 109482240\n",
      "2023-03-16 18:12:36 [INFO ]  Epoch:    0 [      0/     80 ( 0%)]\tLoss: 0.6366\t Data Time: 0.00s\tTrain Time: 0.39s\n",
      "2023-03-16 18:12:37 [INFO ]  Loss, accuracy and DTO: 3.611530 0.300000 0.860236\n",
      "2023-03-16 18:12:37 [INFO ]  Evaluation at Epoch 0\n",
      "2023-03-16 18:12:48 [INFO ]  Validation accuracy: 30.00\tmacro_fscore: 23.08\tmicro_fscore: 30.00\tTPR_GAP: 50.00\tFPR_GAP: 158.11\tPPR_GAP: 158.11\t\n",
      "2023-03-16 18:12:48 [INFO ]  Test accuracy: 27.83\tmacro_fscore: 21.77\tmicro_fscore: 27.83\tTPR_GAP: 0.00\tFPR_GAP: 0.00\tPPR_GAP: 0.00\t\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-03-16 18:12:52 [INFO ]  MLP( \n",
      "2023-03-16 18:12:52 [INFO ]    (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]    (AF): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]    (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]    (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:52 [INFO ]      (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]      (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]      (2): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]      (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]      (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]      (5): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]    )\n",
      "2023-03-16 18:12:52 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:52 [INFO ]  )\n",
      "2023-03-16 18:12:52 [INFO ]  Total number of parameters: 321602 \n",
      "\n",
      "2023-03-16 18:12:52 [INFO ]  BERTClassifier( \n",
      "2023-03-16 18:12:52 [INFO ]    (bert): BertModel(\n",
      "2023-03-16 18:12:52 [INFO ]      (embeddings): BertEmbeddings(\n",
      "2023-03-16 18:12:52 [INFO ]        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "2023-03-16 18:12:52 [INFO ]        (position_embeddings): Embedding(512, 768)\n",
      "2023-03-16 18:12:52 [INFO ]        (token_type_embeddings): Embedding(2, 768)\n",
      "2023-03-16 18:12:52 [INFO ]        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]        (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]      )\n",
      "2023-03-16 18:12:52 [INFO ]      (encoder): BertEncoder(\n",
      "2023-03-16 18:12:52 [INFO ]        (layer): ModuleList(\n",
      "2023-03-16 18:12:52 [INFO ]          (0): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (1): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (2): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (3): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (4): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (5): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (6): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (7): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (8): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (9): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (10): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]          (11): BertLayer(\n",
      "2023-03-16 18:12:52 [INFO ]            (attention): BertAttention(\n",
      "2023-03-16 18:12:52 [INFO ]              (self): BertSelfAttention(\n",
      "2023-03-16 18:12:52 [INFO ]                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]              (output): BertSelfOutput(\n",
      "2023-03-16 18:12:52 [INFO ]                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]                (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]              )\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (intermediate): BertIntermediate(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (intermediate_act_fn): GELUActivation()\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]            (output): BertOutput(\n",
      "2023-03-16 18:12:52 [INFO ]              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "2023-03-16 18:12:52 [INFO ]              (dropout): Dropout(p=0.1, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]            )\n",
      "2023-03-16 18:12:52 [INFO ]          )\n",
      "2023-03-16 18:12:52 [INFO ]        )\n",
      "2023-03-16 18:12:52 [INFO ]      )\n",
      "2023-03-16 18:12:52 [INFO ]      (pooler): BertPooler(\n",
      "2023-03-16 18:12:52 [INFO ]        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]        (activation): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]      )\n",
      "2023-03-16 18:12:52 [INFO ]    )\n",
      "2023-03-16 18:12:52 [INFO ]    (classifier): MLP(\n",
      "2023-03-16 18:12:52 [INFO ]      (output_layer): Linear(in_features=300, out_features=2, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]      (AF): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]      (dropout): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]      (hidden_layers): ModuleList(\n",
      "2023-03-16 18:12:52 [INFO ]        (0): Linear(in_features=768, out_features=300, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]        (1): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]        (2): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]        (3): Linear(in_features=300, out_features=300, bias=True)\n",
      "2023-03-16 18:12:52 [INFO ]        (4): Dropout(p=0, inplace=False)\n",
      "2023-03-16 18:12:52 [INFO ]        (5): Tanh()\n",
      "2023-03-16 18:12:52 [INFO ]      )\n",
      "2023-03-16 18:12:52 [INFO ]      (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:52 [INFO ]    )\n",
      "2023-03-16 18:12:52 [INFO ]    (criterion): CrossEntropyLoss()\n",
      "2023-03-16 18:12:52 [INFO ]  )\n",
      "2023-03-16 18:12:52 [INFO ]  Total number of parameters: 109803842 \n",
      "\n",
      "Number of trainable parameters: 109482240\n",
      "2023-03-16 18:12:52 [INFO ]  Epoch:    0 [      0/     80 ( 0%)]\tLoss: 0.6837\t Data Time: 0.00s\tTrain Time: 0.40s\n",
      "2023-03-16 18:12:53 [INFO ]  Loss, accuracy and DTO: 5.679337 0.250000 1.089725\n",
      "2023-03-16 18:12:53 [INFO ]  Evaluation at Epoch 0\n",
      "2023-03-16 18:13:04 [INFO ]  Validation accuracy: 25.00\tmacro_fscore: 20.00\tmicro_fscore: 25.00\tTPR_GAP: 79.06\tFPR_GAP: 200.00\tPPR_GAP: 158.12\t\n",
      "2023-03-16 18:13:04 [INFO ]  Test accuracy: 27.83\tmacro_fscore: 21.77\tmicro_fscore: 27.83\tTPR_GAP: 0.00\tFPR_GAP: 0.00\tPPR_GAP: 0.00\t\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "kf = KFold(n_splits=5, random_state=state.base_seed, shuffle=True)\n",
    "\n",
    "for i, (train_index, dev_index) in enumerate(kf.split(X)):\n",
    "    state.opt.train_generator.dataset.X, state.opt.dev_generator.dataset.X = X[train_index], X[dev_index]\n",
    "    state.opt.train_generator.dataset.y, state.opt.dev_generator.dataset.y = y[train_index], y[dev_index]\n",
    "    state.opt.train_generator.dataset.protected_label, state.opt.dev_generator.dataset.protected_label = protected_label[train_index], protected_label[dev_index]\n",
    "    \n",
    "    if state.encoder_architecture==\"BERT\":\n",
    "        state.opt.train_generator.dataset.token_type_ids, state.opt.dev_generator.dataset.token_type_ids = token_type_ids[train_index], token_type_ids[dev_index]\n",
    "        state.opt.train_generator.dataset.mask, state.opt.dev_generator.dataset.mask = mask[train_index], mask[dev_index]\n",
    "        \n",
    "    model = networks.get_main_model(state)\n",
    "    model.train_self()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "Thu Mar 16 18:10:07 2023       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  NVIDIA GeForce ...  Off  | 00000000:02:00.0 Off |                  N/A |\n",
      "| 21%   37C    P2    58W / 250W |  10903MiB / 11178MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "|    0   N/A  N/A   4133858      C   .../fairlib_env38/bin/python    10901MiB |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "fairlib_demo.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "fairlib_env38",
   "language": "python",
   "name": "fairlib_env38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "eb4eca40f7710e7c7146430b0424b585ee7a07b7e7498958627feae1c8ad8261"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0598af4dcad946c3b5b27dc73dafda3a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_101c33c843274092888ae0b593e11084",
      "max": 39,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_267c639152c247bfbb01672e66107207",
      "value": 39
     }
    },
    "05c0c0bae60a4c958f6571576dfcd470": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "07824111751947f1af0e3f3409cf8897": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "08b65a34ffbf43fa94aa81bdfa88a839": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "0f5858dbd88d4a8f95d74f8c2f3df4ab": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "101c33c843274092888ae0b593e11084": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "155d359ac2d749169a0d52c591f0d8ea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1f572cf5213c400f86b91e4cb9c5ab03": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_223a07676d284ddc9f88eb557535be33",
       "IPY_MODEL_6d8e56dae7c74b51870b8f205f9c2a51",
       "IPY_MODEL_30564c39e4fc42f9a62391eb4ccbeef5"
      ],
      "layout": "IPY_MODEL_5cf1fb49370f458aa4b34f1d4654c3c0"
     }
    },
    "223a07676d284ddc9f88eb557535be33": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9f28af0718b04e7489b8d81c554b8f0a",
      "placeholder": "​",
      "style": "IPY_MODEL_56054bc9fdae4f4aa8d1dd3b2230349e",
      "value": "100%"
     }
    },
    "267c639152c247bfbb01672e66107207": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2821e713f31b46efa2e90d699e0dc83c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2980b492e74549938e9f0fb78237923d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5456c00ca1a44bf3b7bece88b94250d3",
      "placeholder": "​",
      "style": "IPY_MODEL_f490accfcad641b19c214d9ea5bc422d",
      "value": " 39/39 [00:05&lt;00:00,  7.37it/s]"
     }
    },
    "29a861a0df0549bcba58054ef6558903": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "30564c39e4fc42f9a62391eb4ccbeef5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7bc47b609afe4320a0c754fde01ba508",
      "placeholder": "​",
      "style": "IPY_MODEL_29a861a0df0549bcba58054ef6558903",
      "value": " 39/39 [00:05&lt;00:00,  8.66it/s]"
     }
    },
    "37c4e040cfea4f8c948c358014f0cd00": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_74be6f341a894c7682aad6de3b63c4cc",
       "IPY_MODEL_b9a94eeed9d241a4be13e1ab3b2540d8",
       "IPY_MODEL_88d78f31f08b48608809cd0a211a1fdb"
      ],
      "layout": "IPY_MODEL_155d359ac2d749169a0d52c591f0d8ea"
     }
    },
    "3c29777eb0e64033896044125f33ca3b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "54535b16f2d3462bad55b950d081f572": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5456c00ca1a44bf3b7bece88b94250d3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56054bc9fdae4f4aa8d1dd3b2230349e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5ae45a0c9008496fb0e4347da96546d5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a43ba65c1600446f899efa4e5372f0ad",
      "placeholder": "​",
      "style": "IPY_MODEL_c2d89f3eb92e40b684686fac9636537a",
      "value": "100%"
     }
    },
    "5cf1fb49370f458aa4b34f1d4654c3c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "686ff88b3e02409db49eef590197e51f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5ae45a0c9008496fb0e4347da96546d5",
       "IPY_MODEL_9dd2932943e24383a4cff1cc6df0fad4",
       "IPY_MODEL_c9f3ba4aa2c2405faf9ecb026b233379"
      ],
      "layout": "IPY_MODEL_0f5858dbd88d4a8f95d74f8c2f3df4ab"
     }
    },
    "6d8e56dae7c74b51870b8f205f9c2a51": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c8f1fe9931524c4cb344fd8551e8d918",
      "max": 39,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c3ad77793bd342308f942eeaf7c6db5e",
      "value": 39
     }
    },
    "7041c8004d7f4c2b9f22b200f60394bd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_842e1f5c6d43472b9cb8690e3d2b6bc0",
      "placeholder": "​",
      "style": "IPY_MODEL_76c8de3762d948679a2241e3651476a5",
      "value": "100%"
     }
    },
    "74be6f341a894c7682aad6de3b63c4cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_79aacc7e95bd41d1a7491d300ce44408",
      "placeholder": "​",
      "style": "IPY_MODEL_a993fd47b6a24182a0769207fe5896ff",
      "value": "100%"
     }
    },
    "76c8de3762d948679a2241e3651476a5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "79aacc7e95bd41d1a7491d300ce44408": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7bc47b609afe4320a0c754fde01ba508": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "842e1f5c6d43472b9cb8690e3d2b6bc0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "88d78f31f08b48608809cd0a211a1fdb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a99b784af7c747cb92b304e024cb45e7",
      "placeholder": "​",
      "style": "IPY_MODEL_a826fe75b14e466d9fbff54cd4fd78c7",
      "value": " 39/39 [00:04&lt;00:00,  9.36it/s]"
     }
    },
    "9dd2932943e24383a4cff1cc6df0fad4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2821e713f31b46efa2e90d699e0dc83c",
      "max": 39,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_07824111751947f1af0e3f3409cf8897",
      "value": 39
     }
    },
    "9f28af0718b04e7489b8d81c554b8f0a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a43ba65c1600446f899efa4e5372f0ad": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a826fe75b14e466d9fbff54cd4fd78c7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a993fd47b6a24182a0769207fe5896ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a99b784af7c747cb92b304e024cb45e7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b5e8af5d11954cdbbdb77bdb205e31ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7041c8004d7f4c2b9f22b200f60394bd",
       "IPY_MODEL_0598af4dcad946c3b5b27dc73dafda3a",
       "IPY_MODEL_2980b492e74549938e9f0fb78237923d"
      ],
      "layout": "IPY_MODEL_3c29777eb0e64033896044125f33ca3b"
     }
    },
    "b9a94eeed9d241a4be13e1ab3b2540d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_05c0c0bae60a4c958f6571576dfcd470",
      "max": 39,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_08b65a34ffbf43fa94aa81bdfa88a839",
      "value": 39
     }
    },
    "be8088868f094bee8ad0ca7a1e74e122": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c2d89f3eb92e40b684686fac9636537a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c3ad77793bd342308f942eeaf7c6db5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c8f1fe9931524c4cb344fd8551e8d918": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c9f3ba4aa2c2405faf9ecb026b233379": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_54535b16f2d3462bad55b950d081f572",
      "placeholder": "​",
      "style": "IPY_MODEL_be8088868f094bee8ad0ca7a1e74e122",
      "value": " 39/39 [00:05&lt;00:00,  7.89it/s]"
     }
    },
    "f490accfcad641b19c214d9ea5bc422d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
